Redundancy Rates of Slepian-Wolf Coding∗

The results of Shannon theory are asymptotic and do not reflect the finite codeword lengths used in practice. Recent progress in design of distributed source codes, along with the need for distributed compression in sensor networks, has made it crucial to understand how quickly distributed communication systems can approach their ultimate limits. Slepian and Wolf considered the distributed encoding of length-n sequences x and y. If y is available at the decoder, then as n increases x can be encoded losslessly at rates arbitrarily close to the conditional entropy H(X|Y ). However, for any finite n there is a positive probability that x and y are not jointly typical, and so x cannot be decoded correctly. We examine a Bernoulli setup, where x is generated by passing y through a binary symmetric correlation channel. We prove that the finite n requires us to increase the rate above the conditional entropy by K( )/ √ n, where is the probability of error. We also study the cost of universality in Slepian-Wolf coding, and propose a universal variable rate scheme wherein the encoder for x receives PY = 1 n ∑ i yi. For PY < 0.5, our redundancy rate is K ′( )/ √ n above the empirical conditional entropy. When |PY −0.5| = O(n−1/6), K ′( ) = Ω(n1/6), and another scheme with redundancy rate O(n−1/3) should be used. Our results indicate that the penalties for finite n and unknown statistics can be large, especially for PY ≈ 0.5.